[file_packager.py] Add --export-es6 to file_packager.py #24737

lkwinta · 2025-07-18T11:19:20Z

I have added modularize option flag to file_packager.py script, like in main js generator
I think it will be helpful since in modern world of react it is inconvinent to add generated files as <script/> tags.
This pull request introduces a new --export-es6 option to the file_packager.py tool, enabling the generation of modularized JavaScript output. It enables to import generated JS loading stub into ES6 environment. The main module which will take effect of script execution is now passed as script argument.

Use scenario:

emcc -s MODULARIZE -s EXPORT_NAME=MainModule ...
file_packager.py --export-es6 ...

import MainModule from 'main_module_wasm'

// two datafiles that will be loaded at runtime with module
import { default as loadDataFile_specialName } from 'data_file_preload' 
import loadDataFile from `data_file_preload2`

var preMod = {
   // i. e. locateFile
}

var mod = MainModule(preMod)
mod.then(async (module) =>  {
    await loadDataFile(module);
    await loadDataFile_specialName(module;

    // module has now loaded data file in VFS
});

Fixes: #24504

grzanka · 2025-07-22T20:23:13Z

@sbc100 could you please take a look ?

tools/file_packager.py

sbc100 · 2025-07-22T21:14:59Z

Can you explain a little more in the PR description exactly what --modularize is doing here? It looks like its generating an ES6 module from the data files, is that right? How is the generated module supposed to interact with the program into which the files are to be loaded?

lkwinta · 2025-07-22T21:40:10Z

Can you explain a little more in the PR description exactly what --modularize is doing here? It looks like its generating an ES6 module from the data files, is that right? How is the generated module supposed to interact with the program into which the files are to be loaded?

I have added some use case description.

As described in the related issue we are just trying to make the dynamic loading of large necessary dependencies inside web workers run from react application. Together with XHR lazy loading this seems to be only valid solution to our problem that might solve our problem.

tools/file_packager.py

lkwinta · 2025-08-02T09:47:02Z

@sbc100 what do you think about it now?

tools/file_packager.py

sbc100

Looks like we still need a test for this.

And might be worth adding to the changelog.

Otherwise lgtm.

lkwinta · 2025-08-04T21:09:49Z

Looks like we still need a test for this.

And might be worth adding to the changelog.

Otherwise lgtm.

I have added two simple tests - from what I have seen file_packager.py tests aren't really extensive.

test/test_other.py

ChangeLog.md

test/test_other.py

tools/file_packager.py

lkwinta · 2025-08-08T10:42:24Z

I manage to solve the issue, apperantly we do not have any tests with --no-node support? Is it possible to run test from web context at all to test if fetch works insteads of nodes method?

Now happy path works but promise is not rejected where it should be most likely - there is were I need that PR: #24871

@sbc100

fix python format apply more ruff fixes fix pipeline revert file to the most actual version and apply changes add modularize to options class add utilization of EXPORT_NAME with modularize option simplify declaration of the modularize function arg documentation address @sbc100 review Fix codegen new line formatting Add file_packager tests and changelog mention fix some stylistic issues gichange file_packager modularize test simplyfi tests fix test rename to export_es6 async function promise behaviour cleanup test address @sbc100 review simplify the test rename promise handlers introduce createRequire in es6 mode alternative way to import require

sbc100 · 2025-08-12T19:54:38Z

test/test_other.py

+        .catch((cause) => Promise.reject(cause))
+        .then(() => {
+          module._test_fun();
+        }


I don't think you need the .catch here. Doing return loadDataFile(.. should be enough to reject the promise if loadDataFile fails/rejects.

Also, I think just using top-level-await might be more clear here:

var module = await loadModule(); await loadDataFile(module): module._test_fun();

I will apply that later, I have changed some logic to be more universal and to allow catching thrown error, take a look please

I wonder, if I shouldn't add some throw error method to clear run dependencies - or maybe generate separate stub that allows to do that if we want ourselfs in external catch block

I wonder, if I shouldn't add some throw error method to clear run dependencies - or maybe generate separate stub that allows to do that if we want ourselfs in external catch block

It could be other PR tho, I have just observed anoying behaviour that when it fails, main module constantly spams run dependencies message, being able to clean those to graceful escape error would be nice

I'm not sure that removing the run dependency is the way to fix that.. since that would result in the module running. If a dependency fails we want to block the running forever I think.. not just ignore it. But I think the existing code might not do that :(

test/test_other.py

tools/file_packager.py

sbc100 · 2025-08-12T20:11:20Z

tools/file_packager.py

+
+  if not options.export_es6:
+    ret += '''
+  (async () => {'''


Why is async being added here for the non-export-es6 case?

Because I wanted to wait for the loadPackage which was made async in one of yours PR chain

tools/file_packager.py

sbc100 · 2025-08-12T20:14:18Z

tools/file_packager.py

@@ -1126,9 +1152,15 @@ def generate_js(data_target, data_files, metadata):
  else:
    ret += '''
    }
-    loadPackage(%s);\n''' % json.dumps(metadata)
+    await loadPackage(%s);\n''' % json.dumps(metadata)


If this is the very end of the function then you can just do return loadPackage I think?

(doing return instead of await might also mean you don't need to make the outer function async).

sbc100 · 2025-08-12T20:22:49Z

test/test_other.py

+        .catch((cause) => Promise.reject(cause))
+        .then(() => {
+          module._test_fun();
+        }


I'm not sure that removing the run dependency is the way to fix that.. since that would result in the module running. If a dependency fails we want to block the running forever I think.. not just ignore it. But I think the existing code might not do that :(

sbc100

lgtm % final comments

sbc100 · 2025-08-12T21:58:39Z

tools/file_packager.py

+            } else {
+              fetched = data;
+            }
+          })%s;


It looks like there is no current error handling here, for this fetch. It looks like the code is trying to do a pre-fetch and maybe we can just ignore failures (like we currently do) for the pre-fetch?

In that case this block could be reverted

sbc100 · 2025-08-12T21:59:47Z

tools/file_packager.py

+    catch_case = '''
+          .catch((error) => {
+            loadDataReject(error);
+          })'''


How about doing:

catch_handler = '' if export_es6: catch_handler = ''' ... '''

Then you don't need repeated (catch_case if options.export_es6 else '') below.

sbc100 · 2025-08-12T22:01:52Z

tools/file_packager.py

        }
      };

      var files = metadata['files'];
      for (var i = 0; i < files.length; ++i) {
        new DataRequest(files[i]['start'], files[i]['end'], files[i]['audio'] || 0).open('GET', files[i]['filename']);
-      }\n''' % (create_preloaded if options.use_preload_plugins else create_data)
+      }\n''' % (create_preloaded if options.use_preload_plugins else create_data, ready_promise if options.export_es6 else '')


How about:

finish_handler = create_preloaded if options.use_preload_plugins else create_data if export_es6: finish_handler += '\nloadDataResolve();'

Then use finish_handler here in the string template.

lkwinta mentioned this pull request Jul 18, 2025

Using standalone file_packager #24504

Open

lkwinta force-pushed the file_packager_modularize branch from c4a662f to deac6ec Compare July 22, 2025 20:00

sbc100 reviewed Jul 22, 2025

View reviewed changes

tools/file_packager.py Outdated Show resolved Hide resolved

sbc100 reviewed Jul 22, 2025

View reviewed changes

tools/file_packager.py Outdated Show resolved Hide resolved

sbc100 changed the title ~~[file_packager.py] Add modularize to file_packager.py~~ [file_packager.py] Add --modularize to file_packager.py Jul 22, 2025

lkwinta requested a review from sbc100 July 22, 2025 21:29

sbc100 reviewed Jul 22, 2025

View reviewed changes

tools/file_packager.py Outdated Show resolved Hide resolved

sbc100 reviewed Jul 22, 2025

View reviewed changes

tools/file_packager.py Outdated Show resolved Hide resolved

lkwinta requested a review from sbc100 July 23, 2025 11:58

lkwinta force-pushed the file_packager_modularize branch 2 times, most recently from 1b74c3e to 05f9693 Compare July 23, 2025 21:41

lkwinta force-pushed the file_packager_modularize branch from 47dcb43 to e2b3fc9 Compare August 2, 2025 09:56